Handwritten keyboardless entry computer system

ABSTRACT

A keyboardless entry computer system includes a transparent input screen that generates positional information when contacted by a stylus, and a display screen mounted physically below the input screen such that a character that is displayed can be seen below the input screen. The system includes a computer that has been programmed to compile the positional information into Strokes, to calculate Stroke characteristics, and then compare the Stroke characteristics with those stored in a database in order to recognize the symbol drawn by the stylus. Key features of the system are: (1) transparent position sensing subsystem; (2) underlying display on which to mimic drawing of sensed positions and to show characters or symbols; (3) means to convert sensed positions first into plotted Points and then into recognized characters or symbols; and (4) means to &#34;learn&#34; to associate sensed input positions with a character or symbol.

REFERENCE TO RELATED CASE

This application is a continuation of Ser. No. 08/190,745 filed on Jan.31, 1994 now abandoned, which is a continuation of Ser No. 07/775,167filed on Oct. 11, 1991 now U.S. Pat. No. 5,297,216, which is adivisional of Ser. No. 07/523,447 filed on May 14, 1990 now U.S. Pat.No. 5,157,737, which is a continuation of Ser. No. 07/029,772 filed onMar. 24, 1987 now U.S. Pat. No. 4,972,496, which is a continuation inpart of Ser. No. 06/889,513 filed on Jul. 25, 1986 now abandoned.

REFERENCE TO DISCLOSURE DOCUMENT

Reference is hereby made to Disclosure Document No. 144,644, filed Jan.14, 1986.

FIELD OF THE INVENTION

The present invention relates generally to a keyboardless input systemto a computer, and when combined with a central processing unit, to akeyboardless entry computer system. More particularly, the presentinvention relates to an information storage, manipulation and transferdevice on which text, data, computer commands and functions are enteredby writing alphanumeric or any other characters and symbols by hand witha penlike stylus on an Input/Output (I/O) screen. In a preferredembodiment the I/O screen includes a transparent touch screenincorporated over a substantially flat output display. The presentinvention in its preferred embodiment is a self-contained computersystem but can also function as a peripheral to a host computer.

DESCRIPTION OF THE PRIOR ART

Large amounts of information and sophisticated applications software arenow available on conventional keyboard computers. The utility of thisinformation and of application software could be greatly increased iftext and data could be entered and applications software manipulated bywriting in a normal fashion directly on a flat display. Thus, there is aneed to allow the utility of computer technology to be extended for useby non-keyboard oriented individuals. There is also a need for aportable computer system that is lightweight, reliable, accurate,inexpensive and permits use while standing or walking. One way to reduceexpense and size and increase utility is to employ a keyboardless entrysystem, such as a touch screen. However, this type of input device doesnot easily allow accurate detailed input within a real time frameworkwith high resolution in a manner which is familiar and natural to theuser.

Many positioning technologies can be used to meet the requirements ofthe position sensing input technology. Essentially these requirementsinclude accuracy, resolution and speed. The technologies include:mechanical, electrostatic, electromagnetic, acoustic, optical, andinertial. The desire in this system is to have its use as similar aspossible to writing with pen or pencil on paper. One problem isproximity--a pen on paper only leaves a trail when actually in contact.Many of these technologies require an additional "pen down" sensor whichis awkward to use in many commercial pens. Another problem is writingangle--a pen leaves the same trail independent of writing angle. Many ofthese technologies have the position detector displaced from the pentip, so pen angle causes erroneous displacements. Beyond these generalproblems, each technology has numerous advantages and disadvantages in(1) the pen: size, weight, shape and whether it needs to be poweredand/or wired, and (2) the writing surface: transparency, smoothness,"feel", and whether or not physical contact is needed (as opposed topressure transmitted through overlaying sheets of paper).

A number of self-contained devices for viewing and processing largeamounts of information are known. Most employ optical, magnetic orsolid-state electronic storage means to store data. Illustrative of thisbody of art is U.S. Pat. No. 4,159,417 to Rubincam which discloses aportable electronic book configured to provide selective page by pagecall-up of large amounts of digital data and displays it on a flat,solid-state screen. The preferred embodiment in the Rubincam patent usesan insertable holographic card, which may contain several hundred pagesof text in digital form, as the main storage means. Rubincam's device,however, does not allow new information or text to be entered ormanipulated.

In U.S. Pat. No. 4,016,542 to Azure an electronic data collection systemis disclosed which employs a solid state Random Access Memory (RAM) forits primary memory. This patent, which discloses a conventional keyboardfor data entry, is directed to a hand-held portable data storage andtransmission system, as well as an LED display and various Input/Output(I/O) connectors.

U.S. Pat. No. 3,487,731 to Frank discloses a means of convertinghandwriting into character data through the use of a computer system.The disclosed invention is based on matrix pattern matching and does notemploy any coincident display technology.

U.S. Pat. No. 4,491,960 to Brown shows a handwritten symbol recognitionsystem in which an array of image Points, in the form of a raster linesampling, is converted into segment-oriented lists which are filteredand compressed to obtain topologic features which are then analyzed witha logic tree decision mechanism.

U.S. Pat. No. 4,262,281 to Buckle et al. discloses a handwritingrecognition device. The disclosed embodiment is for use with a hostcomputer and does not employ coincident display technology.

U.S. Pat. No. 4,475,239 to Van Raamsdonk discloses a text editingapparatus. The '239 patent calls for the use of paper as a medium forthe entering of editing functions and requires a conventional keyboardfor the inputting of text.

U.S. Pat. No. 4,521,909 to Wang shows a dual level pattern recognitionsystem. The system is designed for use with an optical instrument.

U.S. Pat. No. 4,520,357 to Castleberry et al. discloses an electroscopicinformation display and entry system with writing stylus. The disclosedembodiment does not claim to have the speed or accuracy to enablehandwritten character recognition.

Additional prior art which discloses portable electronic devices thatprovide large amounts of various types of stored information includeU.S. Pat. Nos. 4,218,760 to Levy; 4,4115,486 to Laine; and 3,932,859 toKriakides et al. The Levy and Kriakides et al. patents pertain toelectronic dictionaries, while the Laine patent discloses a programmabletelevision reminder system. None of these devices disclose the use of ahandwritten input.

In U.S. Pat. Nos. 4,071,691, 4,129,747, 4,198,539, 4,293,734, 4,302,011,4,353,552, 4,371,746 and 4,430,917 to William Pepper, Jr. variousmethods or machine-human interfaces using finger touch are disclosed.The preferred embodiments in each of these inventions lack sufficientspeed and resolution to allow handwritten character recognition with astylus and are designed for other purposes. U.S. Pat. No. 4,318,096 toPepper teaches the use of a conductive stylus. The '096 patent pertainsto graphic design and allows line width and line intensity to vary byapplying pressure on the stylus with the results displayed on aconventional CRT screen. U.S. Pat. No. 3,699,439 to Turner and U.S. Pat.No. 4,055,726 to Turner et al. disclose two methods for electronicposition sensing through the use of a probe.

SUMMARY OF THE INVENTION

The invention comprises a unique keyboardless computer system which hasthe ability to recognize and display Handwritten Symbols and cause thecomputer to display Font Symbols and, if desired, to execute editingfunctions pursuant to Editing Symbols, quickly, easily and at reasonablecost.

The invention constitutes a computer housing with a flat display panelon which a user may "write" with a stylus, a capability to recognizeHandwritten Symbols written on the panel with the Stylus and convertthem to displayed Font Symbols and/or to execute Editing Functions withEditing Symbols, all with a minimum of technical complexity for theuser.

A further feature of this invention is that once the keyboardless,portable computer is loaded with the desired information andapplications software, information and software can be used andresponded to without requiring skills or knowledge related tostate-of-the-art computers or other data source.

The ease-of-use of the input technology of the present inventionenhances the utility of the computer for keyboard oriented individuals.The portability of the present invention also allows it to be used inapplications and settings in which portable keyboard computers areawkward, difficult or impossible to use. For example, a multiplicity ofblank, fully or partly completed forms may be stored in the portablecomputer memory. In a hospital, "sheets" of patient data can be storedin the memory of the portable computer, called up by a nurse as thenurse makes rounds and relevant data, such as blood pressure,temperature, etc., can then be entered manually with a stylus. Thesecorrected or expanded forms can then be down-loaded into a centralcomputer memory.

The requirements of the position sensing input technology are accuracy(Point to Point), resolution (absolute position) and speed (Points perunit time) to adequately define the written Stroke for recognitionanalysis. For the recognition apparatus and methods presently used, asdescribed below, the present minimum requirements are: accuracy of 0.005inch, resolution of 0.015 inch, and speed of 150 Points per second. Thisaccuracy allows a 1/4" high writing line with over 10 raw input Pointsalong a Stroke of a small letter. The resolution provides positioning ofthe symbol to within two pixels on a present display of 640 pixels to 9inches. The speed permits about 50 raw input Points for a rapidlywritten single letter (1/3 second).

One embodiment of the present invention comprises a transparent inputscreen. As the user writes alphanumeric or other characters or symbolson the input screen, the character is represented as a stream of Pointsemulating written input with pen on paper. Once the discretealphanumeric and other characters or symbols are complete, they aretranslated into computer text or computer commands that can be displayedon a display screen at a location that is preferably beneath the area onthe input screen where they were entered. The embodiment also comprisesa pattern recognition algorithm which allows the translation of anywritten character or symbol, such as ideographs and scientific symbols,into computer text.

In a particular, presently preferred embodiment, a keyboardless computeraccording to the present invention is configured as a manipulation anddisplay device comprising a transparent touch screen and associatedelectronics placed over an 80 column by 25 line or larger displayscreen; a stylus for entry of data; a microprocessor and storage means;artificial intelligence/pattern recognition software and editingsoftware; and a battery power system; and other I/O means.

As used herein, "Handwritten Symbols" are any symbols capable of beinghandwritten and having communicative effect. By way of example, and notlimitation, numbers, letters, "Kanji" (Japanese ideograms) or otherlanguage symbols, editing symbols and engineering, scientific,architectural and mathematical symbols are Handwritten Symbols. Otherexamples of Handwritten Symbols are free-hand drawings or signatures orany other such written information uniquely configured by a particularwriter. Handwritten Symbols may also include Editing Symbols (definedbelow).

As used herein, "Font Symbols" are computer-generated symbols which aredisplayed in a predetermined font format. By way of example and notlimitation, alphanumeric symbols may be Font Symbols and displayed innumerous font formats. Japanese or Chinese "ideograms" may also be FontSymbols, as may be engineering, scientific, mathematical, architecturalor other such characters. Other examples of Font Symbols include anyform which can be stored and displayed by a computer, e.g., a drawing ofa car or a house.

As used herein, an "Editing Symbol" is a symbol (such as a caret,horizontal line, short vertical line, long vertical line, etc.) which isintended, when recognized, to cause the computer to execute a particularEditing Function (defined below), such as insert text (caret), deletetext (horizontal line), delete a letter (short vertical line) or move amargin (long vertical line), to list a few representative examples.

"Editing Function" means any computer-generated text editing operation,such as by way of example and not limitation, insert text, delete text,move text and substitute text. Some primary Editing Functions are listedon paged 40 and 41 below.

OBJECTS OF THE INVENTION

It is therefore a primary object of the present invention to provideimproved methods and apparatus for providing a keyboardless computer onwhich usual computer functions are performed by writing in a normalmanner with a pen-like stylus on an input screen placed directly over aflat display.

The keyboardless computer provided is ideally configured for use bynon-keyboard oriented individuals, by keyboard individuals for whom theutility of the computer is enhanced, and in various settings andapplications in which keyboard entry is awkward or impossible.

It is also an object of this invention to provide a means wherebycomputer-based information and applications software can be loaded intoa portable device for later viewing, manipulation of text and data, andadding new text and data in a normal handwriting mode. Thereafter theuser may transmit this computer text to another computer, a similardevice, an external electronic storage device, a hard copy printer, orthrough a telecommunications system. Yet another object of thusinvention is to provide a computer capable of recognizing HandwrittenSymbols with a high degree of accuracy and of "learning" individualstyles of handwriting.

A further object of this invention is to provide a portable keyboardlesscomputer in which data and commands are input with the use of a stylus.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic system block diagram of the present invention;

FIG. 2 is a perspective view of the housing containing the operatingelements of the invention;

FIG. 2A is an enlarged portion of FIG. 2, with parts removed to show thepositional relationship between the touch input screen and the displayscreen.

FIG. 3 is a schematic diagrammatic view of the input screen, stylus andassociated electronics;

FIG. 4 is an overall schematic system block diagram of the apparatus ofa keyboardless entry computer system according to the present invention;

FIG. 5 is a schematic block diagram depicting the movement of datawithin the system when modified by handwritten characters and commands;

FIG. 6 is an overall system block diagram depicting the hierarchy ofsoftware used to operate the system;

FIG. 7 is a generalized block diagram of the character and patternrecognition algorithm.

FIGS. 8A and 8B together are a detailed block diagram of the patternrecognition algorithm.

FIG. 9 is a schematic block diagram of the Stroke characterizationsubroutine.

FIG. 10 is a top plan view of a screen illustrating the "initializing"of the database for Handwritten Symbols.

FIGS. 11A through 11I are a series of top plan views of screensdepicting the operation of a text editing system.

FIGS. 12A through 12G are a series of top plan views of screensdepicting the operation of a data entry system.

FIG. 13 is a generalized block diagram of the Linus Editor.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

With reference now to the figures, wherein like numerals indicate likeelements throughout the several views, and in particular with referenceto FIG. 1, an overall block diagram of a portable handwritten,keyboardless entry computer system 10 is depicted. The complete computersystem is encased in a housing 12, indicated graphically by the dashedline, and includes a conventional, general purpose digital microcomputer14, described in greater detail hereinbelow. Input information isprovided to microcomputer 14 by stylus 16 "writing" on writing or inputscreen 18. Stylus 16 (FIG. 2) is connected to the computer of system 10with wire 17 (FIG. 2). As stylus 16 "writes" on input screen 18, aplurality of locating signals representative of a plurality ofcorresponding positional coordinates are transmitted to microcomputer14. Microcomputer 14 has been programmed in accordance with a computerprogram described hereinbelow, to recognize the stream of locatingsignals and to store these signals in a computer memory. The programmedmicrocomputer 14 also provides a corresponding plurality of displaysignals to a display screen 20. Both input screen 18 and display screen20 are described in greater detail hereinbelow.

Referring now to FIG. 2, there is shown a perspective view ofkeyboardless computer system 10 embodying the features of the presentinvention. Keyboardless computer system 10 is contained in housing 12,which is a rectangular enclosed casing having a sloped top surface 22with a multi-line, solid state display area 24. In one embodiment,keyboardless computer system 10 is a portable unit having dimensions ofup to about 16"×16"×4" and weighing up to about 15 pounds. Input screen18 is depicted in FIG. 2A as being positioned over display screen 20. Inthis example, display screen 20 displays a plurality of horizontal lines25 with the following indicia: ##STR1## Handwritten entries are madeabove each line 25. The distance or space between two lines 25, denoted26, is used by the system to normalize all distances, and lines 25themselves serve as a reference axis or base line.

Below display area 24 on top surface 22 is a key input section 26comprised of a plurality of "Softkeys" 28. Softkeys 28 can be programmedby the operator for any purpose, such as to enter computer commands.Exemplary commands for Softkeys 28 are "store," "recall," and "delete."In addition, Softkeys 28 can be used to switch between differentprograms or between modes (e.g. data entry mode and edit mode). However,Softkeys 28 are optional and are used to supplement the input obtainedby handwriting the entries. Stylus 16, used for writing input data andcommands in display area 24, also is used to activate the selectedSoftkey 28. An ON-OFF switch 30 is positioned on the side of housing 12adjacent to Softkeys 28. A data output or peripheral connector 31 islocated on the upper right side of housing 12.

Input screen 18 can be a conventional resistive type touch screen inwhich a voltage is applied to the screen edges and a stylus detects thevoltage at the touched location. The writing surface is a transparentmaterial, typically glass, coated with a thin, uniform, conductive layer(presently, vacuum deposited indium tin oxide). Vertical bus bars orconducting strips (not shown) are used along the two sides to apply thereference voltage to determine the "X" coordinates of the stylusposition and horizontal bus bars or conducting strips (not shown) areused along the bottom and top to apply the reference voltage todetermine the "Y" coordinates of the stylus position. In thisembodiment, stylus 16 is merely an electric probe that, when physicallyin contact with the conductive layer, detects the local voltage at thePoint of contact, which will vary with the distance from the conductingstrips or bus bars. With the origin at the Point of voltage application,the X, Y coordinates are inversely proportional to the impressedvoltage. Stylus 16 must make good contact to minimize adding resistancethat would lower the voltage detected, and thus add an erroneousdistance increment. In a presently preferred embodiment, a soft graphitetip is used. The voltage is conducted from the pen through a wire, suchas wire 17 in FIG. 2, to an analog to digital converter for use in thecomputations described below. The stylus may be a charged "pen" asdescribed herein, a light pen as is well known in the art, or any otherhand-held device which can outline Handwritten Symbols on a screen.

An example of a conventional electrostatic screen is disclosed in theaforementioned U.S. Pat. No. 4,318,096 Pepper patent. This resistivetype screen has the advantage that the interference caused by the user'shand touching the screen is minimized.

Both horizontal and vertical position sensing is provided by alternatelyswitching the voltage impressed on a conductive layer between the pairsof horizontal and vertical bus bars by an interface and multiplexercontrolled by a microcomputer or microcontroller. In one commerciallyavailable input or touch screen, the bus bars are broken into a seriesof short strips with diodes to prevent the horizontal strips fromshorting out vertical strips and vice versa. This technique is used in acommercially available touch screen from Touch Technologies ofAnnapolis, Md. and Elographics of Oak Ridge, Tenn.

Referring now to FIG. 3, an alternate embodiment of a low power positionsensing novel input screen 33 is described in greater detail. Inputscreen 33 is also for determining an X,Y position on an electricallyresistive plate 34. A stylus 35 containing a voltage source, such asbattery 36 or a voltage transmitted to stylus 35 from an external sourcesuch as the system power supply, is used to touch screen 34 and apply avoltage at the touched position. When the touched position is charged bystylus 35 with a positive voltage with respect to a plurality of platemeasurement Points 37, the voltages at these Points will vary with thedistance to the pen position, such as position X₁, Y₁, indicated at 38.These voltages are sequentially measured in the X and Y directions byusing conventional means, such as disclosed in the aforementioned priorart patents. In FIG. 3 these means are a conventionalinterface/multiplexer 42. A conventional Analog-to-Digital converter 43converts the detected voltages into a digital signal. A microcontroller44 receives the digital signal, performs standard checks to insure thesignal's numerical value is "valid" (e.g. is within the possible rangeof voltages), and then converts the voltages to X and Y distances in themanner described herein. Microcontroller 44 is conventional, but couldbe replaced by a system computer. Microcontroller 44 provides a digitalsignal representative of the X and Y distances to measuring Point 38 toan output port 46. Port 46 can be a conventional RS 232 port.Alternatively, microcontroller 44 could translate Point X₁, Y₁ to anyother reference Point, such as a Point on base line 25 (FIG. 2).

As long as there is no contact by stylus 35 at position 38 or any otherposition on plate 34, no current flows and power consumption is minimal.An incident measurement of the voltage at the measurement Points mayoccur by using ramped voltage at the positioning Point and timing whenthe measuring Point voltage exceeds a preset back voltage.

The scope of this invention covers the following options for the inputor touch screens 18 and 33: resistive plate 34 or its equivalent forscreen 18 can be transparent or translucent and the position Point canbe made by a stylus or a finger of the user, or a connecting Point of anoverlapping conductive screen (such as the commercially available touchscreens from Touch Technology, Annapolis, Md.). Input screens 18 and 33can be a physical solid surface which is transparent or translucent andcan be glass or plastic such as Mylar. The surface can be coated with aconductive/resistive substance like indium tin oxide. Other physicalsurfaces can use sound or electromagnetic radiation transmission fromthe touched position to a reference Point or Points and the distance isdetermined by the time delay or phase shift. Alternatively, inputscreens 18 and 33 can use an ethereal or geometric surface defined by anelectromagnetic, optical or sonic field.

Position detection can be accomplished with electrical contact closureby resistive, capacitive or inductive coupling, remote sensing by sonic,electric or magnetic fields or by light (UV, IR, or microwave) scanning.

The advantages of the low power position sensing input invention overother such screens are: 1) the invention makes stand-by powerrequirements minimal; 2) the invention eliminates distortion due toopposing parallel "bus" bars in conventional touch screens; and 3) whena ramped voltage is employed, the invention eliminates the need for anA/D chip which is a major cost factor in state-of-the-art touch screentechnology.

The coefficient of friction of the screen 18 is desirably selected to be"rough" enough to offer some resistance to the movement of stylus 16 onthe screen. If the screen were too smooth, the stylus would slide tooeasily and would be difficult to control.

Reference is now made to FIG. 4 which discloses an overall system blockdiagram of the major electronic circuitry used in the preferredembodiment of the present invention. Microcomputer 14 includes amicroprocessor 50, interconnected to a plurality of other electronicelements by means of data path or bus 52. A Read-Only-Memory (ROM) 54which is programmed with the operating and application programs and abattery powered Random Access Memory (RAM) 56 is connected forbidirectional data flow onto bus 52. Microprocessor 50 may be aconventional single-chip eight-bit or sixteen-bit device which functionsto execute the fixed control programs residing in ROM 54, and furtherreceives control programs from and provides control signals to the otherelectronic elements via bus 52. Microprocessor 50 may be of the typecommercially designated Z80 (manufactured by Zilog Microcomputers ofCupertino, Calif.), of a type 8088 device (manufactured by Intel Corp.of Santa Clara, Calif.) or any similar or more powerful microprocessor.ROM 54 may be of the type 2564 or 4764, both manufactured by TexasInstruments of Dallas, Tex. The storage capacity of RAM 56 is determinedin part by the sizes of the application programs, the operating programand the database. As discussed below, RAM 56 may be of the static SRAMor dynamic DRAM type. The primary requirements of RAM 56 are that ithave sufficient storage capacity and that it require a minimum of inputpower.

A battery 58, such as a lithium battery, provides power for making thememory of RAM 56 non-volatile for extended periods of time. A batterypack 60 containing the well-known rechargeable types of batteries isused to provide the various voltage levels required by the otherelectronic elements of microcomputer 14.

Alternately, the storage function of RAM 56 may be served by anon-volatile device which requires no power for maintaining storage,such as an electronically erasable and reprogrammable memory (EEPROM),or devices using magnetic bubbles or capacitance. State-of-the-art diskor tape may also be used for mass storage. Suitable bubble memorydevices include types 7110 and 7114 which have storage capacities of 1megabit and 4 megabits respectively. (Both are manufactured by IntelCorp.). Furthermore, it is possible to use a single integrated circuitchip which includes microprocessor 50, at least part of ROM 54 and atleast part of RAM 56.

Also connected to bus 52 is an EIA RS-232 serial interface 62 whichprovides a means for inputting and outputting data. Data is provided tobus 52, (usually to RAM 56) by interconnecting an external data sourceto RS-232 port 62 directly to the microprocessor 50 and other elementsof the microcomputer 14. Offloading data from RAM 56 can also be done bymicroprocessor 50 to an external computer, other data gathering device,a mass data storage device (e.g. floppy and hard disk drives) or anelectronic telecommunications system. In like manner data can becommunicated through port 62 to a printer (not shown) frominterconnecting bus 52.

Stylus 16 is used to write on input screen 18 and to cause thegeneration of X, Y coordinate information by conventional touch screeninterface electronics circuitry. The coordinate information iscommunicated via the bus 52 for control use by system 10. The solidstate display 20 consisting of a multi-line display--illustratively 80columns by 25 lines--is interconnected to bus 52 through a displayinterface 66. The fundamental requirements for the display are that itbe substantially flat and sufficiently thin for use in the presentinvention. The display may be of the following types: scanning typessuch as a cathode ray tube, projected types such as a rear-viewprojector, light emitting array of Points types (e.g.,electroluminescent or plasma discharge) and light blocking array ofPoints types (e.g., liquid crystal displays, solid state PLTZ ormagneto-optical). In addition, it is preferable that the display becompatible with input screen 18 in size, configuration and transparency,and that both be low power consuming types.

The X,Y coordinates for this invention are input to keyboardlesscomputer 14 via input screen interface electronics 64 and communicatedvia bus 52 to microprocessor 50 which executes programs stored in ROM 54and RAM 56.

The number of Points (i.e., sets of X,Y coordinates) used in definingeach Handwritten Symbol and the speed at which Points are identified areimportant to the practical utility of the invention. It is desirable touse at least about 100 Points per inch and at least about 100 Points persecond to define Handwritten Symbols. It is to be noted that the morePoints per inch that are identified, the greater the accuracy of thesystem in identifying Handwritten Symbols--however, more Points beingidentified will slow down the speed of identification and require morecomputer memory. Accordingly, a balance will have to be achieved, basedon the size (available memory and processing ability) of the computersystem and the requirement for speed of response and accuracy. For mostpurposes, standards in the range from about 100 Points per inch and persecond to about 200 Points per inch and per second will be suitable.

It is also to be noted that the greater the precision of the system inidentifying the X,Y coordinates of each Point the fewer the number ofPoints needed to be identified per inch and per second to accuratelyidentify Handwritten Symbols. Conversely the less the accuracy, the morePoints that are needed.

Point resolution is needed to place Points where intended, e.g., towrite an editing symbol precisely between the two characters. Ideally,resolution to a single display pixel is desirable. However,operationally, resolution within two displayed pixels is sufficient fora display with 640 pixels in a nine inch horizontal scanline.

When switch 30 (FIG. 2) is positioned to "power on", the basic displaymode is activated and microcomputer 14 (FIG. 4) programmed by theoperating system, causes a menu to be displayed on display screen 20(FIG. 1). The menu presents various software options. A primary softwarefunction, editing, functions in a manner similar to conventional wordprocessing software with the difference being that handwrittencharacters, symbols and commands are interpreted by the system as ifthey were entered from a conventional keyboard. The system is capable oflearning the editing symbols used by a particular writer for functionssuch as indent, insert, delete, move and reformat and translates thosesymbols into digital command functions. Optionally, Softkeys 28 (FIG.2), activated by touching those areas on the input screen with stylus16, function like conventional hard function keys on a computerkeyboard.

The present invention is particularly adapted for use as an interactivescreen editor or word processor. After a writer retrieves a document by(for example) touching the displayed name of an existing file with thestylus or by writing the name of the file on the screen, all usualediting functions can be performed with stylus entry. When the userwishes to change a displayed character or symbol, he may simply writeover the displayed character or symbol and as described hereinbelow thepattern recognition algorithm will translate the written entry intocomputer text. For example, the editing software allows text to beeliminated by simply drawing a line through it and a conventional caretsymbol may be used to change the operating mode to the insert mode. Inthe insert mode, display screen 20 provides additional space for entryof handwritten characters or symbols which are inserted in the textafter the Point where a caret was written in. Text can be moved simplyby placing brackets or other user-defined delimiters around a displayedphrase or word and writing a caret or other user-defined symbol in thearea of the text in which the user wishes this material to appear. Newmargins can be set by drawing vertical lines down the side of thedisplayed text where the new margins should appear.

The basic editor software also allows new documents to be created bysimply writing Handwritten Symbols on the screen. All documents can bestored, changed and communicated in the manner in which these functionsare accomplished on a conventional word processing system with thedifference that these functions are accomplished with handwrittenEditing Symbols on the (optional) screen or by touching the Softkeyswith the stylus. The composite text thus produced and stored can besubsequently offloaded through the RS 232 port 62 (FIG. 4) to anothercomputer, a similar device, an external data gathering device orrecording device, into a printer or through a telecommunications system.

In addition to these major operating modes, a number of ancillaryelements and features add to the utility of the present invention. Aconventional alphanumeric keyboard (not shown) containing a full set ofkeys can be connected to a conventional keyboard interface (not shown)to support the entry of alphanumeric characters. An AC/DC powerconnector may also be used in those applications when portability is notneeded and when needed to meet the power requirements of screentechnologies such as gas plasma displays and electroluminescentdisplays.

In actual use the keyboardless computer can function in any applicationor environment in which handwritten input translated into computer textis useful or necessary. Illustratively, the device can function as a newgeneration word processor, or for use in fields such as sales, nursing,inventory control, census taking, claims adjusting, to name just a fewof the many uses of the invention, or as a general learning and testingdevice in education. Since the pattern recognition software can learnand translate into computer text from languages which are not made up ofa small or limited set of alphanumeric characters (e.g., Japanese,Korean, and Chinese), it has particular utility for word processing andtelecommunications in these languages.

In the practice of this invention, it is particularly desirable to use asingle computer screen to display any initial forms, Font Symbols orother displays to be edited, and to create a nearby "window" of blankspace where Handwritten Symbols are to be written, displayed andidentified, and where the Font Symbols corresponding to the HandwrittenSymbols are to be displayed. In this way, the user can view the textbeing edited and the proposed insert or change without significantmovement (if any) of the head and eyes. This is illustrated in FIGS. 11Ato 11D. This feature of the invention (proximity on one screen of textto be edited and the window into which new text is to be handwritten) isvery important to the simple, rapid, comfortable use of the invention.

In a preferred embodiment of this invention, the system "learns" thehandwriting of a particular user prior to actual use. For example, ifusing the Roman alphabet, the twenty-six letters of the alphabet and thenumerals from 0 to 9 would be inserted into the database. Punctuationsymbols, such as periods, commas, question marks, colons, semi-colons,hyphens and the like could also be inserted. There is virtually no limitto the Handwritten Symbols which can be recognized and stored in thedatabase. Of course, the computer will have to store a suitable array ofFont Symbols for conversion of the Handwritten Symbols. Different setsof Font Symbols could be created and stored in the permanent memory ofthe computer, as in ROM chip 54. For example, in English language usage,a chip could contain one (or more) fonts of numbers and letters,suitable punctuation symbols and appropriate mathematical symbols. Otherchips could have stored Font Symbols for the Arabic, Cyrillic or Greekalphabets, Japanese, Chinese or Korean "Kanji", symbols for use byarchitects or engineers, or chemical symbols (e.g., benzene rings andthe like).

In FIG. 10, one of a series of learning screens is displayed and theuser is prompted to write the numbers 0 through 4. The computer willattempt to match the written numbers with the existing database (ifany). If it cannot be matched because there is no existing database orbecause there is a poor match with an existing database, the characteris added to the database. This learning process continues until all ofthe alphanumeric (or other) characters and symbols to be used areentered into the database. The system has the capability of storingmultiple Stroke characterization databases for systems used by more thanone user. The existence of a unique Stroke characterization database foreach user has the further advanage of making the writing angleirrelevant. As a result, the invention is adaptable to all handwritingstyles and is usable by right-handed and left-handed persons. Onefeature may desirably be incorporated into the apparatus of theinvention to accommodate left-handed and right-handed persons. Thisfeature is a receptacle (not shown) for the stylus connector on eitherside of housing 12, so that the stylus 16 may be connected on the leftside for left-handed persons and on the right side for right-handedpersons.

FIG. 10 also provides an example of the use of "Softkeys". In additionto the input line, a variety of Softkeys appear. Each Softkeycorresponds to a function that can be performed by the system. In orderto execute the funtion, the user merely touches the indicated Point withthe pen. The Softkey will then appear in reverse video and the selectedfunction is performed. There are numerous advantages to Softkeys overtraditional function keys. Some of the more significant of these arethat the user is no longer required to memorize what function keyperforms what function; the need for keyboard overlays is eliminated:and different Softkeys can be made available (displayed and madeoperational) at different Points within a program.

FIGS. 11A to 11I demonstrates some of the simplifications in wordprocessing made possible through the use of this invention. In FIG. 11Aa standard screen of text is displayed. The user of the keyboardlessentry system decides that additional information needs to be added anddraws an insert symbol (e.g., caret) on the screen at the desiredposition. A data entry "window" then appears. (FIG. 11B). The text iswritten in as Handwritten Symbols (FIG. 11C), matched (converted to FontSymbols) (FIG. 11D), and then inserted (FIG. 11E). The operatorreconsiders the addition and draws a horizontal line through the newmaterial. (FIG. 11F). It is immediately erased. (FIG. 11G). Next, theoperator decides that a larger right-hand margin would be moreappropriate for the text. A vertical line is drawn on the screen (FIG.11H) and the margin is automatically adjusted (FIG. 11I).

A generalized block diagram of the editing process is provided in FIG.13 and a description of that figure appears hereinbelow.

FIGS. 12A-12G illustrate how a blank form may be used for a hospitalpatient. The user of the system first calls up the proper blank form(FIG. 12A). This may be done, for example, by touching an appropriateSoftkey. The area where the information, in this case a pulseobservation, is to be inserted is touched with the pen (FIG. 12B). Afterthe desired location is highlighted, a "window" appears directly belowthe space where the observation is to be recorded (FIG. 12C). The nursethen touches the pen on the match box which appears highlighted (FIG.12D). The software then matches the handwritten input to thecorresponding Font Symbols and displays the result (FIG. 12E). If thereis an accurate match, the "insert" block is touched (FIG. 12F), and thenew observation is added to the patient's records (FIG. 12G). Thismechanism is clearly applicable to a wide variety of "blank forms" inwhich data is inserted into a form or corrected. For example, it couldbe used to correct or update financial information in a spreadsheetprogram. All such applications are within the purview of this invention.Other information can be recorded in the same manner.

The reason for using a black background and white letters for the newlyentered Font Symbols is to facilitate checking the accuracy of theinputted character. Although this is preferable, it is not essential anda white background and black letters is also acceptable.

The ability to create a window and input data on the same screen and inphysical proximity to the text being edited or space for data to beinput is an important feature of this invention, for it permits ease andspeed in the use of the invention. The user's eye may focus on the spacewhere the data will be inserted and the ability to contemporaneouslydisplay Handwritten Symbols and the corresponding Font Symbols makes iteasy to see errors, when the system "misreads" a Handwritten Symbol, andthen correct errors quickly and easily.

Referring first to FIG. 5, the overall operation and functioning of thepattern recognition software will now be described. When the operatingsystem calls the pattern recognition program, the program begins interminal 75 where a number of variables and counters are initialized.The software then proceeds to decision diamond 76 where the programdetermines if stylus 16 (FIG. 2) is in contact with input screen 18(FIG. 2A). The system provides a "pen down" signal, as shown inprocessing box 78, as well as the X,Y coordinate voltages as locatingsignals, as described above. Microcomputer 14 (FIG. 4), using thesoftware according to the present invention, converts the X,Y coordinatelocating signals into Stroke characteristics using programs stored inROM 54 (FIG. 4), or a separate microcomputer can do the conversion, suchas microcontroller 44. If a pen down signal is received, the softwareproceeds to processing box 80 where the individual locating signals arecombined into "Strokes," a Stroke being defined as the Point locatingsignals produced between a "pen down" signal and a "pen up" signal.

The system then calculates a transform, as described below, for eachPoint, transforming the Point coordinates from the X,Y cartesiancoordinate system to a relational coordinate system. The software nextproceeds to processing box 82, where it compares the Stroke withpreviously entered Strokes accumulated into a database, and determinesif the Stroke is represented by a symbol in the database. If a match isfound (if the Font Symbol represented by the Strokes is recognized), asindicated in decision diamond 84, microprocessor 50 (FIG. 4) causes thesymbol to be sent to display screen 20 (FIG. 4) as indicated inprocessing box 86. If a match is not found, microprocessor 50 (FIG. 4)causes a message to be displayed, as indicated in processing box 88,which requests further input from stylus on input screen 18 (FIG. 4) byeither flashing an entry which is close to a match or a non-recognitionsymbol.

As mentioned above, the software compares the Stroke characteristics ofeach Handwritten Symbol to data entries previously stored in a database.In a preferred embodiment, the database is arranged into sections ofcharacters or symbols by the number of Strokes needed to make thecharacter or symbol. Within each section, the entries are randomlyarranged at first, but after use, as explained herein, the mostfrequently used entries "rise" to the top of the database. It should benoted that each user will have his or her own particular style ofwriting a Handwritten Symbol and that each Handwritten Symbol may have anumber of different variations.

For example, many people write the lower case letter "h" using a singleStroke. They do this by starting the pen on the writing tablet at aPoint where they wish to place the top of the letter, drawing a verticalline downwardly to the base line, then without removing the pen from thepaper, proceeding back up to the midPoint of the previously drawnvertical line, over to the right and down to the base line when the penis picked up from the paper. On the other hand, these same people maydraw the upper case letter "H" using two Strokes. They do this bydrawing the left hand vertical line and horizontal line as is done forthe lower case "h", picking the pen up from the tablet, and then drawingthe right hand vertical line. Appendix I displays the data of the Strokedata Points for these two letters as the data is stored in memory afterhaving been generated by an embodiment of the present invention.

As shown in Appendix I, the letter "h" as drawn at one particular timeby one user has one Stroke (ns=1) with 20 Points (np=20) and x and ycoordinate characteristics for the minimum, mean and maximum normalizedvalues (1/80th of a line width) as follows: -17 and -6; 0 and 18; and 19and 60, respectively. The values in the first vertical column are thePoint-to-Point slopes, normalized to 360°/256. The values in the secondvertical column are the Point-to-Point average vertical positions abovethe base line, normalized to 1/80 of the line width. A typical linewidth is about 0.4 inches.

Referring now to FIG. 6, a software hierarchy of programs is depicted.At the top, overseeing the entire operation of computer system 10 (FIG.1), is an operating system as indicated by box 90. Applications programsshown in boxes 92 and 94, residing in RAM 56 (FIG. 4) and ROM 54 (FIG.4) can be executed by microprocessor 50 (FIG. 4) under control of theoperating systems. When a Handwritten Character is required or isindicated by an interrupt, handwriting recognition software 96 iscalled. A first subroutine, indicated in box 98, encodes the X,Ycoordinates into Strokes. The characteristics of the Strokes are thendefined by a subroutine 100 followed by comparisons of the Strokes witha database that has been loaded from ROM 54 (FIG. 4) into RAM 56 (FIG.4). The comparison is made by a subroutine 102. When the operatingsystem is in the "learning" mode, the database is updated with the newStroke data and symbols, as indicated in box 104. Similarly, apreviously stored document can be edited by applications program 92 byusing edit function 94 as called by the operator, who provides theinstructions as input using the subroutines 98, 100 and 102 ofhandwriting recognition program 92.

Referring now also to FIG. 7, operating system 90 (FIG. 6) executes theHandwritten Character recognition software 96 (FIG. 6) by accepting asinput the X,Y coordinate Points, depicted in box 110, of the position ofstylus 16 (FIG. 2) on input screen 18 (FIG. 2) and encodes these Pointsinto Strokes as depicted in box 112. The program then characterizes theStrokes by some description set, such as considering the length,curvature, slope, and position of the Stroke, as depicted in box 114. Inbox 116 the best comparison is then found of the characterized Stroke orsequence of Strokes with those in the database. If a sufficiently closematch is found, the character is identified in box 118 and the databaseentry is swapped with the entry above it as shown in box 120. In thisway, the most frequently identified characters will "rise" to the top ofthe database and the overall system performance, as measured in time tofind a match, will be increased. If a match is not found, the user canadd to the bottom of the database, as indicated in box 122.

With reference now to FIGS. 8 and 9, a flowchart of the computer programto recognize a particular Stroke sequence is set forth. The computerprogram begins in terminal 150 and proceeds to process the X,Y voltagesfrom processing box 152, the voltages having been converted to a digitalsignal. The program then proceeds to decision box 154 where the programdetermines whether the pen or stylus 16 (FIG. 2) is out of contact withinput screen 18. This determination is made by both the X voltage andthe Y voltage being zero. If the program determines that the pen is up,then the Stroke is determined as having been completed and the programbranches to decision box 156. In decision box 156, the programdetermines whether there are less than three Points in the Stroke and ifso the program branches to decision box 158. In decision box 158, theprogram determines whether there are zero Points in the Stroke. If thereare zero Points in the Stroke, then the program loops back to thebeginning of processing box 152 where another set of Points is read. Ifthe Point counter (incremented in processing box 164) indicates thatthere are more than zero Points, the program branches to processing box172. In processing box 172 the Stroke is identified as a dot and itsheight above the base line (HABL) is calculated in processing box 173.From processing box 173 the program proceeds to processing box 171.

However, if the pen down signal is received, the program branches toprocessing box 160 where the voltages are scaled to determine thecoordinate Point using the following formulas:

    X=a.sub.1 v.sub.1 +b.sub.1

    Y=a.sub.2 v.sub.2 +b.sub.2

The constants a₁ and b₁ are scaling parameters that are determined fromcalibrating the input surface of the particular display.

Once the voltages are scaled, the program proceeds to decision diamond162 where the program determines whether it is an erroneous Point. Thisis done by comparing the distance between Points and eliminating a Pointif the distance is to great (greater than 0.10 inches is presentlyused.) On the other hand, a Point is also eliminated if the Points aretoo close together. Points are presently thinned out if they are within0.015 inches.

The comparison problem that exists for the first Point is resolved bydetermining if a Point is the first Point after a pen is down and thenthat Point is used only to check the next Point which is accepted,assuming that that Point is within the maximum distance (0.10 inches).

If the distance between Points is determined as being outside the twocriteria, the program drops the Point and branches back to the top ofprocessing box 152 to read another pair of coordinate Point voltages.

On the other hand, if the Points fall within the criteria, the programcontinues to processing box 164 where a Point counter is incremented tokeep track of the number of Points. This number is used in decisiondiamond 156, as mentioned hereinabove. The program then continues toprocessing box 166 where the Points are smoothed according to any one ofa number of formulas. Smoothing is used to minimize noise fromdigitization, from erratic hand motion and from electronic noise. Thesimplest smoothing technique is a multiple Point average which resultsin calculating new Points (x_(j) 'y_(j) ') as follows: ##EQU1## Andsimilarly for y_(j), smoothed over Points n₁ -n₂.

Another simple method is called the running weighted average method andutilizes the following formula:

    x.sub.j '=αx.sub.j +(1-α)x.sub.j '-1

Alpha is a weighting constant that is usually positive (and less thanone) and has been used at 0.25. The summations have been taken with n₂minus n₁ equal to one. A third method involves what is called a splinefit wherein the following formula is used:

    x.sub.j '=(x.sub.j-1 +4x.sub.j +x.sub.j+1)

Any of the foregoing methods can be applied either before or afterfiltering. The filtering is done so as to reduce the number of inputPoints and to space data so that difference and/or angle calculationscan be made within acceptable random error bounds. A simple process ofthinning a sequence of Points by excluding the acceptance of subsequentPoints within a set distance of the previously accepted Points has beenfound to be an effective filter.

From processing box 166, the program proceeds to processing box 168where the Point is stored in an array that is incremented for each newPoint since the last pen down signal. Thus, an addressable array ofPoints is created for each sequence of Points obtained from a pen downto a pen up signal. This sequence of Points is called a Stroke. Fromprocessing box 168, the program loops back to the top of processing box152 where another Point is obtained until a pen up signal ends theStroke.

In decision diamond 156, a determination was made as to whether therewere less than three Points in a Stroke. By definition, if there arethree or more Points in a Stroke, the Stroke is a line and not a dot. Ifthere are three or more Points in the Stroke, the program branches tosubroutine box 170. In subroutine box 170, discussed in greater detailhereinbelow with respect to FIG. 9, the Stroke is characterized as toits slope and base line height.

As can be seen from the foregoing, the segmentation of the stream ofcoordinate Points into a Stroke is based primarily on the determiningwhen stylus 16 is "up" or not in contact with the surface of inputscreen 18. Alternatively, a stream of Points can be segmented to formStrokes on the basis of other considerations. For example they can besegmented based upon changes in a locally calculated curvature or upon alarge local curvature. Local curvature is calculated by the change indistance along the input coordinates divided into the change in slope.This produces radius of curvature. When the radius of curvature changesrapidly with respect to distance along the input coordinates, or if theradius is too small, then a segmentation Stroke is assumed to end, and anew Stroke begun. Further segmentation techniques can look at therelative maximum and minimum in one or both coordinates and/or the curvecrossings in the coordinates. However, these latter two methods havebeen determined to be less effective.

Characterizing a Stroke reduces the sequence of coordinates defining theStroke or segment to a set of characteristics that are unique,generalized and minimal. Uniqueness refers to both factors that the samecharacteristics are generated by the same coordinates and that thecharacteristics are sufficient to regenerate an approximation to theoriginal coordinate sequence. The term "generalized" is used to meanthat the characterization is invariant under such transformations sothat the symbols are invariant (e.g., translation and scaling orstretching or small tilt). The scaling of all distances is accomplishedby taking a ratio of the distance to a writing entry line width.

The minimal set of segment characteristics have the following features:

(1) Stroke position: one or more of centroid/average, extent extreme orbeginning and ending Points determined relative to the writing entryline, to previous Strokes, or to character extent or center;

(2) Stroke shape is characterized by one or more of average slope,change in slope (which is a measure of average curvature) and/or achange in curvature, by sequence of slopes over specific length segmentsor over fractional lengths, or by a gross description of lineardirection or circular completion and opening direction;

(3) And Stroke length as characterized by distance along the curveand/or the extent extremum along the coordinate system.

In one embodiment of the present invention, positioning by centroid,extent extremum, and starting and ending coordinates have beensuccessfully used. The Stroke shape is encoded as a sequence of slopesand vertical positions (relative to Stroke centroid). The Stroke lengthcan be approximated by the number of filtered Points. Alternatively, theaverage curvature can be encoded in total slope change (along withlength), change in starting to ending slope or fitting the slope angleversus length curve for rate of change of slope angle. Additionalcharacteristics that could be used include location of coordinaterelative extrema, curve crossing, cusps, and Stroke direction. Aparticular method used to determine the unique characteristics is setforth below.

1. The numerical values of the Criteria for each Stroke of theHandwritten Symbol are determined.

2. The database values for each stroke of the previously learnedHandwritten Symbol is determined and subtracted from the newlydetermined values respectively.

3. The absolute values of each difference are scaled, to make each ofthe five measurements reasonably equivalent to the others such aslengths scaled to height between lines.

4. The five thus-determined values are added.

5. A predetermined threshold is used as "goodness" test ofrecognition--too high a value and Font Symbols are infrequentlyrecognized and too low a value causes Font Symbols to be misidentified.Thresholds of approximately 1,000 are used initially and then switchedto approximately 100 for improved recognition. If the threshold isexceeded, the comparison is discarded and an error message is createdand displayed.

6. The database is searched to find a numerical minimum difference. Ifthe minimum difference is below the acceptable threshold forrecognition, the corresponding Font Symbol is displayed on the screen orthe command is performed, as the case may be.

It has also been found that the preferred classification of a Stroke isa continuous one, rather than one that is grossly discrete. For example,determining a slope by angle in 256 directions rather than in 8 isdesirable. Other non-continuous classifications can includebars/arches/hooks, number and closure of cusps or horizontal or verticalStrokes.

From subroutine 170, the program proceeds to processing box 171 whereboth an individual Stroke and one or more preceding Strokes are comparedwith a database entry that is stored in RAM 56 (FIG. 4).

This comparison initially begins with three eliminating questions thatare asked by the program in decision diamonds 174, 176, and 178. In eachcase, if the database entry is eliminated, the program proceeds to aprocessing box 180 where the address of the next data entry is receivedand from which the program loops back to the top of processing box 171.In decision diamond 174, the first eliminator is asked by seeking if thenumber of Strokes are different. If the number of Strokes are the same,the program proceeds to decision diamond 176 where the average HeightAbove Base Line (HABL) is calculated and compared with the HABL of thedata entry. The entry is eliminated if the difference in the averageHABL's is greater than one-half the height of the entry line. From anegative determination in decision diamond 176, the program proceeds todecision diamond 178 where the number of Points per Stroke are comparedand the database entry is eliminated if the difference in number ofPoints is greater than ten. This determination varies from that made indecision diamond 174 because it is concerned only with the number ofPoints for each Stroke. However, in decision diamond 174, certainletters, such as the capital letters "E" and "A", have more than oneStroke per letter.

If a data entry is not eliminated by decision diamond 178, then theprogram proceeds to processing box 182 where the program calculates agauge to be used to determine the closeness of the match between theselected entry in the database and the drawn Stroke. A presentlypreferred gauge is the sum of the absolute values of the differencesbetween the Stroke values and the database entry values of:

a) distances or lengths n units of 1/80th of the line height (e.g.,space 26, FIG. 2); and

b) the slopes in units of 1/256th of 360° over all the Points along thediagonal of the comparison matrix.

Alternatively, Dynamic Programming Techniques can be used to optimizethe comparison using off-diagonal elements as well.

From processing box 182, the program proceeds to decision diamond 186where a match is determined. In actuality, a match is determined by (theapplication of an arbitrary) gauge (maximum allowable variance), whichis the sum of absolute values of the differences between the enteredStroke characterization and that of the stored database entry. Inprocessing box 183, the lower of the present gauge and the previouslower gauge is saved as the best match. The program then goes todecision diamond 184 where a determination is made whether the presententry is the last database entry. If it is not, the program branches toprocessing box 180 where the next entry is selected. If it is the lastentry, the program proceeds to decision diamond 185 where adetermination of a match is made on the basis of the gauge being below apredetermined threshold. This threshold is set by the user based onexperience with the system.

If no match is obtained, the program branches to decision diamond 188where a determination is made whether all Strokes have been checked. Ifthe last Stroke has been checked, then the present Stroke is compared insequence with a previous Stroke to all two Stroke entries. As in thecomparison with all one Stroke dictionary entries, the best fitcomparison for all entered Strokes is the recognized symbol or sequenceof symbols.

However, if the last Stroke has been read and there still is not amatch, then the program proceeds to processing box 190 where a questionis displayed on display screen 20 asking the user if a new Font Symbolshould be added to the database. The user responds and that response isused in decision diamond 192. Either the Stroke sequence is added to thedatabase in processing box 194 and the program branches back to the topof processing box 152, or the program branches immediately to the top ofprocessing box 152.

On the other hand, if a match is determined in decision diamond 186, theprogram branches to processing box 195 where the program shuffles thedatabase by interchanging the serial location of the matched entry withthe entry above it. The program then proceeds to processing box 196where the program zeros the Point counter and the increment counter. Theprogram next proceeds to processing box 198 where the matched andcharacterized Stroke or Strokes are displayed by the computer as theidentified Font Symbol. This display is located at the position in whichthe entry was made on input screen 18 (FIG. 2).

From processing box 198, the program proceeds to processing box 200where the program can act on any commands which it has interpreted. Analternative characterization of the Stroke uses the Points themselvesrather than the length, scope, curvature and position.

With reference now to FIG. 9, this Stroke characterization is depictedin greater detail. Stroke characterization subroutine 170 essentiallyperforms a mathematical transformation of each Point on a Point-by-Pointbasis to transform the Points from an X,Y Cartesian coordinate system toone in which the coordinates are the normalized slope of each Point andthe normalized height of each Point above the base line (HABL).

Subroutine 170 first calculates the Point to Point slope in processingbox 220 and then calculates the height of each Point above the base linein processing box 222. The slope and HABL of each Point are thennormalized respectively to 1/256th of 2 Pi and to 1/80th of the width ofthe entry line in processing box 224. From processing box 224, thesystem proceeds to processing box 226 where the calculated normalizedvalues for each Point are stored in an addressable array. The subroutinethen returns to the program through terminal 228.

When the comparison is made between each Stroke and the stored values,the comparison is made by the normalized Point slope and Point heightabove base line. As mentioned above, a match is determined by anarbitrary gauge which is the sum of absolute values of the differencesbetween the written Stroke and the stored or dictionary Stroke. Thesystem learns by adding new Strokes to the dictionary database. Once thedatabase fills up, those Font Symbols that are infrequently used arereplaced by new entries.

In a working embodiment of the present invention, the algorithmsuccessfully identified upper and lower case letters and numbers whenwritten discretely from one another. For Handwritten Symbols that arewritten such that they are continuous, direct extrapolation wouldrequire searching a database sequentially for one, two, three, etc.Stroke symbols and looking for the best fit. Upon identification of aStroke fit, a "new" letter is tentatively recognized, except that thenext few Strokes are analyzed to check if they change the previoussymbol for a better fit. For example, two Strokes that have beenidentified as "ones" would be combined and changed to the capital letter"H" once a cross bar was identified.

The system design demonstrated by FIGS. 7 to 9 could easily be coded byone with ordinary skill in the art of computer programming into almostany computer language. The source code listings for one applicationprogram utilizing the disclosed invention is included as Appendix II.The software in Appendix II is written in Microsoft Basic, a commoncomputer language available for virtually all microcomputers andoperating systems. The program is a complete text editing demonstration,which takes advantage of many of the key features of this invention andshows the improvements that can be made upon traditional word processingsystems through the utilization of this invention.

Program lines 2600 to 4000 contain the character recognition subroutinewhich includes the software code necessary to get X and Y coordinates.This section of the program corresponds to FIG. 8.

Program lines 2600 to 2699 make up a subroutine designed to obtain the Xand Y coordinates of a given Point. This code corresponds to boxes 152,154, 160, 162, 164, 166 and 168 of FIG. 8.

Program lines 3000 to 3339 constitute a Point and Stroke analysis andcharacterization routine embodying boxes 156, 158, 170, 172 and 173 ofFIG. 8.

Program lines 3700 to 3790 make up a subroutine designed to compare theanalyzed Strokes to a Stroke database. These program lines embody boxes171 to 184 of FIG. 8.

Program lines 3810 to 3980 make up a subroutine which is designed tolearn a new character. This code embodies boxes 186, 188, 190, 192 and194 of FIG. 8.

Program lines 3060 to 3273 make up a subroutine designed for Strokecharacterization purposes. This section of the code corresponds to FIG.9.

Program lines 3060 to 3095 are used to calculate Point-to-Point slopesand embody box 220.

Program lines 3058, 3241 and 3262 are used to calculate a height abovebaseline (HABL) and correspond to box 222 of FIG. 9.

Programs lines 3253, 3270-3273 are used to normalize the Point heightand slope and correspond to box 224 of FIG. 9.

Program line 3253 is used to store the height above baseline andembodies box 226 of FIG. 9.

The foregoing program can be stored in the memory of a microcomputer ofmicroprocessor with a requirement of approximately 25K of machinememory, so that it can be seen that the use of the program does not useup a lot of expensive memory and is relatively fast in executing theprogram's operation. If the program is written in a language other thanBasic, requiring less memory, such as assembly language, the size of theprogram can be made smaller.

Boxes 195 to 200 of FIG. 8 appear in logical places throughout the code.

A dictionary of the variables of the relevant code section is includedas Appendix III.

With reference now to FIG. 13, a flowchart for the editing software("Editor") demonstrated by FIGS. 11A to 11I and described above isdepicted. Once the Editor is loaded into the system (box 229), controlof the screen is returned to the system. The system then proceeds in thenormal manner described above to acquire Points and display them (box230), convert the Points into Strokes (box 231), characterize eachStroke (box 232), and attempt to match the Stroke or Strokes with thedatabase (box 233). In processing box 234, the system sends eachHandwritten Symbol to the Editor to interpret and execute a command ifnecessary. At decision diamond 235, the Editor determines whether theHandwritten Symbol is an Editing Symbol or a Font Symbol. If thecharacter is determined to be an Editing Symbol, the Editor proceeds toprocessing box 236 where it determines which Editing Symbol has beenentered and executes the Editing Function. If the character isdetermined not to be an Editing Symbol, then the alphanumeric charactercorresponding to the handwritten entry is displayed at processing box237. In an alternate configuration of the Editor, Font Symbols will onlybe accepted when the Editor is in the "Insert Mode." This structureinsures that each Font Symbol is verified before being added to adocument.

The Editor uses a variety of symbols designed to make editing on thesystem similar to, but much more efficient than, traditional editingwith pencil and paper. These functions include, but are not limited to:

DELETE symbol--"₋₋₋₋₋₋₋₋₋₋ " A horizontal line drawn through a characteror characters. The Editor will remove the underlying characters andreformat the text.

ADJUST MARGINS symbol--"|" A vertical line longer then the height of oneline on the display. The Editor will adjust the margin to the indicatedposition and reformat the text.

INSERT symbol--" " A caret drawn at the Point where text is to be added.The Editor displays an input writing line (FIG. 118) and when input isrecognized inserts it into the text.

MARK TEXT symbols--"<" and "> A less than and greater than symbol drawnat the beginning and end of a block of text. The marked text isdisplayed in reverse video and then special block functions can beperformed.

DELETE MARKED TEXT--A delete symbol drawn within marked text will erasethe marked text and reformat.

MOVE MARKED TEXT--An insert symbol drawn anywhere within the text movesthe marked text to the indicated position, deletes it from its originalposition and reformats the text.

REPLACE MARKED TEXT--An insert symbol drawn within the marked textdisplays an input line and replaces the marked text with the inputtedtext.

The Editing Symbols described above can be changed to the particularEditing Symbols preferred by each user, thereby customizing the Editorand preventing new users from having to learn unfamiliar EditingSymbols.

Further modifications and enhancements to the present invention would beobvious to those skilled in the art. For example, the commoncharacteristics of each Font Symbol could be extracted and organizedinto a synthetic symbol. The synthetic symbol's characteristics couldthen be exaggerated to maximize their variance from all other syntheticsymbols. This would create a very compact, optimal database. On theother hand, as an example, a database created by the described preferredembodiment of the invention usually results in two to three differentcharacterizations for each symbol.

The invention has numerous useful applications, almost withoutlimitation. The most obvious applications are text editing and fillingout and modifying forms. Some of the many other applications that maynot come to mind as readily are writing in languages utilizing largenumbers of symbols like Japanese or Chinese; writing in Arabic andsimilar languages made up of a limited number of complex symbols;writing chemical equations, including those involving organic compounds;writing music (a "window" with five parallel lines can be provided formusical applications); writing symbols and codes for graphicmanipulation of data, including the transfer of graphic data to aspreadsheet; in education, as where predetermined questions arepresented on the screens and the answers written in long-hand; as inteaching mathematics, as when numbers are manually inserted in equationsand the equation analyzed to determine the result using those numbers;in CAD/CAM applications involving symbols, geometric shapes and thelike.

Although the invention has been described in terms of a selectedpreferred embodiment encompassing the apparatus and methods aspects of akeyboardless computer system, the invention should not be deemed limitedthereto, since other embodiments and modifications will readily occur toone skilled in the art. ##SPC1##

What is claimed is:
 1. In a device for recognizing and displayinghandwritten symbols, an apparatus, comprising:a display forelectronically displaying predetermined text; a processor coupled tosaid display, which causes the display of a window on said display forentering at least two handwritten symbols at arbitrary locations withinsaid window; and a digitizer, coupled to said display, for digitizingsaid handwritten symbols, said processor, coupled to said digitizer, forcorrelating at least one of said digitized handwritten symbols with atleast one of a plurality of predetermined font symbols to provide one ofsaid plurality of predetermined font symbols as a designated font symbolcorrelated with one of said handwritten symbols so that said processorcauses said designated font symbol to be displayed near said window inresponse to receiving said at least one digitized symbol.
 2. In a devicefor processing information which includes a display device, a digitizer,and a device for recognizing digitized symbols, for displayingrecognized characters on said display device and for executing commandscorresponding to recognized command symbols, a method for inputtingsymbols to said device comprising:digitizing a handwritten symbol usingsaid digitizer and providing a digitized symbol to the device forrecognizing, said digitizer and said display device positioned one overthe other to permit viewing of said display while looking at saiddigitizer; displaying, in response to entry of a handwritten commandsymbol, a window on said display device, said window for entering atleast two handwritten symbols at arbitrary locations within said window;converting at least one of said two handwritten symbols entered in saidwindow to a predetermined font symbol; and displaying said predeterminedfont near said window.
 3. In a device for processing information whichincludes a display device, a digitizer, and a device for recognizingdigitized symbols, for displaying recognized characters on said displaydevice and for executing commands corresponding to recognized commandsymbols, a method for permitting a user to input symbols to said devicewithout a keyboard comprising:digitizing a handwritten symbol using saiddigitizer and providing a digitized symbol to the device forrecognizing, said digitizer and said display device positioned one overthe other to permit viewing of said display while looking at saiddigitizer; displaying predetermined text in at least a first region ofsaid display device, said first region having a first area and a secondarea; receiving a first command symbol drawn in said first area toindicate a first portion of text to be edited; causing a window toappear on said second area of said display device, upon execution of apredetermined command, for entering at least one handwritten symbol atan arbitrary location within said window, said second area being spacedfrom said first area so that said window does not obscure said firstportion of text, said second area being sufficiently close to said firstarea to permit simultaneous viewing of both said first portion of textand said window; and displaying near said window a predetermined fontsymbol corresponding to said at least one handwritten symbol entered insaid window after receiving said at least one handwritten symbol.